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Environment 

Model 


Interchangeable  parts  saves  money 


What  Metrics  are  there  to  measure 
the  Cost  Effectiveness  of  standards 

Implementation? 

Not  much  in  the  literature 

Personal  experience  with  SISO,  SEDRIS, 
HLA,  DIS,  CoT,  DTED,  GPS,  etc.  etc. 

Some  are  good  and  save  time  and  money 

Some  are  real  dogs 


How  to  analyze  interoperability? 


Step  1 .  Reduce  Interoperability  to  a 
Programmers  Task 


Application 


<IOP„  Libraries 


Presentation 


Session 


Transport 


Network 


Data  link 


Bkv&ical 


ICDls  Memos. 
E-mails.  P2P 


IOP  Standard 

a  r 


 IOP  2  Libraries 


> 


Lower 
ISO  NIodel 
Lavers 


< 


J 


v. 


Presentation 


Session 


Transport 


Network 


Data  link 


Plivsic 


leaders 


tneaimiR 


trailers 


Step2:  Analyze  Interoperability 

Architecture 


Fig.  2.  -  Five  Interoperable  Systems  without  standards 
Requires  (N-l)N  interoperability  tasks  (N=5) 


Step3 :  Reduce  The  interoperability 
Tasks  by  Introducing  Standards 


Fig  3  -  Five  Interoperable  Systems  types  with  standards 


Reduces  the  number  of  tasks  from 
N*(N-1)  to  N 


Step  4:  Calculate  the  value  of 
introducing  a  standard 

•  (ISPV)  Interoperability  Standards  Project  Value 

•  (IoPT)  Interoperability  programming  task 

•  (SER)  Standards  Effectiveness  ratio 

•  (SCR)  Standards  Complexity  ratio 


Value  of 
the 

Standard 


Cost  with 
no  standard 


minus 


Cost  with  a 
standard 


ISPV  =  N(N-l)*IoPT  -  {  N*SCR  +  N*(N-l)*(l-SER)}*IoPT. 


Step5:  Introduce  a  comparative  metric 
Interface  Standards  Project  Effectiveness 

(ISPE) 

•  (ISP  Vi)  Value  of  the  ideal  standard  with 

SER=SCR=1 

•  ISPVi=  (N(N-1)  -  N}*IoPTs 

•  (ISPE)  Standards  Project  Effectiveness 

•  ISPE  =  ISPV/ISPVi 

•  Which  reduces  to  the  basic  metric  equation: 

ISPE  =  [(N-1)*SER  -SCR]/(N-2) 


Cost  Reduction  Assumption 

A  standard  is  ideal  under  two  assumptions.  These  are: 

•  Standards  Complexity  Ratio  SCR=1;  i.e.  the  task  cost  of 
executing  the  interoperability  task  to  the  standard  is  equal 
to  the  cost  of  the  point  to  point  interoperability  task. 


•  Standards  Effectiveness  Ratio  SER=1  if  there  are  no  latent 


point  to  point  tasks  required  in  addition  to  the  interface  to 


the  standard.. 


Interoperability  Standards  Project 
Effectiveness  (ISPE)  ratio 

ISPE  =  [(N-l)*l  — 1  ]/(N-2)  =1 


In  this  case  we  have  an 
ideal  standard. 


Fig  3  -  Five  Interoperable  Systems  types  with 
standards  for  an  ideal  standard 


Step  6:  To  Evaluate  Real  Standards 

•  Estimate  the  cost  of  the  interoperability 
Programmers  task  (IoPT s) 

•  Estimate  the  standards  Effectiveness  ratio 
(SER) 

•  Estimate  the  standards  Complexity  ratio 
(SCR) 

•  And  do  so  as  a  function  of  standards 
characteristics 


Define  the  Interoperability 
Programmers  Task 
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Future  Tasks 


2010- 

-  Finalize  and  document  the  Standards  Cost 
Effectiveness  (SCE)  prototype 

2011- 

-  Work  with  VMASC  on  “Standards  in  Modeling 
and  Simulation:  Next  Ten  Years”  project  Hon. 
Randy  Forbes 

-  SISO  presentations  and  papers  advertise 
capability 

-  Test  and  work  with  Standards  Program  offices  ( 
CoT,  Ucore,DIS,... 
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Backup  Slides 


Example  Task  Estimation  Formula 


IoPT  =  KDOMpF*F  +  WPARS  +  WEXTpF*F  +  KSF2LpF*N  +  WSF2LpF  *N 

+  KENCpF  *N  +  WFOR  +  WENCpF*N  +  KL2SFpF*N  + 

WL2SFpF  *N  (18) 


Where: 


F 


Number  of  fields  or  individual  data  items  handled  in  the 


standard 


KDOMpF  .lhrs 
WPARS 
WEXTpF  lhr 
KSF2LpF  lhr 
WSF2LpF  2hr 
KENCpF  .lhr 
WFOR 

WENCpF  lhr 
KL2SFpF  lhr 
WL2SFpF 


Knowledge  to  decode  DOM  location  to  IoS  field  name  per  field 
8hrs  Write  Parser  and  message  decoder 
Write  extraction  routine  per  field 

Knowledge  to  understand  IoS  in  terms  of  local  parameter  per  field 
Write  IoS  field  to  local  parameter  translator  per  field 
Knowledge  to  encode  IoS  Filed  names  into  DOM  locations  per  field 
8hrs  Write  a  formatter  that  builds  a  IoS  message  buffer 
Write  encoder  from  IoS  Filed  names  into  DOM  locations  per  field 
Knowledge  to  understand  local  variables  in  Ios  terms  per  field 
2hr  Write  local  variable  to  IoS  symbol  translators  per  field 


I  got  these  values  by  coding  a  test  interface  in  C++  and 
keeping  track  of  my  time  per  coding  task  I  got  726hfS 


SCR 

Complexity  estimation  considerations 

•  IoS  Message  Parser  and  Extract 
Routines 

•  Context  Decoder  Information 
Requirements 

•  Context  Encoder  Information 
Requirements 

•  Programmer  qualifications 


SCR 

Complexity  Estimation 


By  substituting  the  values  estimated  in  this  example  for  the 
lOOfield  case  used  in  section  4  we  estimate  a  task  cost 

IoPTc  =  ,5*F  +  160  +  1*F  +  2*F  +  2  *F  +  .75  *F  +  24  +  3*F  + 
1*F  +  2  *F  =  184  +  12.25*F  =  1409  man  hours 

To  be  consistent  we  should  add  the  same 
programmer  qualification  cost  of  90hrs  to  get  1499hrs 
compared  with  726hrs  for  the  P2P  case.  The  standard 
complexity  ration  is  then 

SCR  =  IOPT/IOPTi  =  1499/726hrs  =  2.06 


Effectiveness  Categories 

•  Incompleteness  (M) 

•  Knowledge  Ambiguities  (O) 

•  Field  Ambiguities  (P) 

•  Undocumented  Extensions  (X) 

•  Subset  extensions  (SXS) 

SER  ={F/(F+M)}*{F/(F+0)}*{F/(F+P)}*{F/(F+X*SXS)}=  84.8%  (16) 

Example  Test  Case:  F=  for  a  100  field  standard,  M=5,  P=5,0=2  and  all 
others  zero. 


Plugging  into  Standards  Effectiveness 

F  ormulas 

for  a  20  node  100  parameter  interface 

Cost  Savings  if  we  had  an  ideal  standard 

ISPVi=  {N(N-l)  -  N}*IoPTs  =  (20*19-20)*726=  13,068hrs 
Effectiveness  of  our  test  examle  standard: 

ISPE  =  [(N-1)*SER  -SCR]/(N-2) 

=  [(20-1)*. 84  -2.06]/(20-2)  =  14/18  =  77% 

Cost  savings  if  we  introduce  our  test  example  standard: 

ISPV  =  ISPE*ISPVi  =  .77*13, 068hr  =  10164hrs 

Cost  to  the  project  because  the  example  standard  is  has 

deficiencies: 


ISPV  -  ISPVi  =  2904hrs  or  1.5  man-years 


